Access control for Hadoop

鍦ㄨ秴澶ц妯℃暟鎹鐞嗙殑浜戣绠楅鍩� Hadoop宸茬粡鎴愪负宸ヤ笟鐣屽拰瀛︽湳鐣岃繘琛屼簯璁＄畻搴旂敤鍜岀爺绌剁殑鏍囧噯骞冲彴銆侶adoop瀹炵幇浜嗗寘鎷垎甯冨紡鏂囦欢绯荤粺HDFS (Hadoop distributed file system)鍜孧apReduce妗嗘灦鍦ㄥ唴鐨勪簯璁＄畻杞欢骞冲彴鐨勫熀纭�灦鏋�骞朵笖鍦ㄥ叾涓婃暣鍚堜簡鍖呮嫭鏁版嵁搴撱�浜戣绠楃鐞嗐�鏁版嵁浠撳簱绛変竴绯诲垪搴旂敤骞冲彴銆備娇鐢℉adoop, 鐢ㄦ埛鍙互鍦ㄥぇ瑙勬ā闆嗙兢骞冲彴涓婂紑鍙慚apReduce绋嬪簭鏉ュ鐞嗘捣閲忔暟鎹�鐢变簬瀹冪殑浣庢垚鏈拰鏄撶敤鎬� Hadoop宸茬粡鎴愪负瓒婃潵瓒婂鐨勫叕鍙哥敤浣滃ぇ瑙勬ā鏁版嵁澶勭悊鐨勯噸瑕佸熀纭�蒋浠躲�

鍦ㄤ簯璁＄畻鐜涓�鏁版嵁瀹夊叏鏄墍鏈夌敤鎴烽兘鎷呭績鐨勪竴涓噸瑕侀棶棰樸�鍩轰簬HDFS鍜孧apReduce妗嗘灦鐨勪簯璁＄畻骞冲彴骞朵笉鑳芥彁渚涘緢濂界殑瀹夊叏鎬с�杩欐槸鐢盚adoop妗嗘灦鑷韩鐨勫畨鍏ㄦ満鍒舵墍鍐冲畾鐨勩�HDFS鍜孧apReduce鐨勮璁′箣鍒濆苟娌℃湁鐗瑰埆鍏虫敞瀹夊叏,浣滀负涓�釜搴曞眰鍒嗗竷寮忔搷浣滅郴缁�瀹冩棤娉曟敮鎸佷綔涓氭墽琛屾椂鐨勬渶灏忔巿鏉�閭ｄ箞鎵�湁鍩轰簬HDFS鍜孧apReduce妗嗘灦涔嬩笂鐨勫簲鐢ㄥ钩鍙颁篃鏃犳硶鎻愪緵寰堝ソ鐨勫绉熸埛瀹夊叏鏀寔銆備负浜嗚鏄庨棶棰樼殑閲嶈鎬�杩欓噷鍋氫釜绫绘瘮銆傚湪缁忓吀Linux绯荤粺涓� ping绋嬪簭鐨勮繍琛岄渶瑕佽鎺堜簣root鏉冮檺,浣嗗疄闄呬笂ping绋嬪簭鍙渶瑕佺敤鍒颁竴涓垱寤簉aw_socket鐨勭壒鏉冦�缁檖ing鎺堜簣root鏉冮檺鎵嶈兘瀹炲湪鏄お杩囦簡,浣嗙粡鍏窵inux kernel骞朵笉鏀寔鏈�皬鎺堟潈,鎵�互杩欎簺杩涚▼缁忓父鎴愪负鏀诲嚮鐨勭洰鏍囥�閭ｄ箞鍥炲埌Hadoop鍒嗗竷寮忕郴缁�涓婂眰鏁颁粨搴旂敤Hive蹇呴』浠uperuser韬唤杩愯浜嶩DFS鍜孧apReduce涔嬩笂,浣嗗疄闄呬笂鐢ㄦ埛鎻愪氦鐨凷QL鎴栬嚜瀹氫箟鍑芥暟寰�線鍙渶瑕佽闂湁闄愮殑搴曞眰璧勬簮銆傛墍浠�鏀诲嚮鑰呮瀬瀹规槗鍒╃敤涓婂眰搴旂敤鏈嶅姟鏉ユ敾鍑诲簳灞侶adoop绯荤粺骞冲彴,浠庤�瀵艰嚧鏁版嵁瀹夊叏鏃犳硶淇濊瘉銆�/p>

鏈�皬鎺堟潈鍘熷垯(the least-privilege principle)鏄璁″閿欑郴缁熺殑涓�釜閲嶈鍘熷垯,涔熸槸璁捐鍒嗗竷寮忔搷浣滅郴缁熷畨鍏ㄧ殑涓�釜閲嶈鍘熷垯銆傛敮鎸佹渶灏忔潈闄愪笉浠呰兘鏄捐憲澧炲己绯荤粺鐨勫畨鍏ㄦ�鍜岀ǔ瀹氭�,鑰屼笖鍙互涓轰笂灞傚簲鐢ㄦ彁渚涙洿濂界殑瀹夊叏鏈嶅姟銆傜己灏戝鏈�皬鏉冮檺鐨勬敮鎸�灏辨棤娉曠洿鎺ヤ娇鐢℉adoop骞冲彴鏉ユ瀯寤哄畨鍏ㄧ殑浜戣绠楁湇鍔°�

Capability[1,2]鏄疄鐜版渶灏忔潈闄愮殑鍩烘湰鏂规硶涔嬩竴,浣嗗畠涓嶈兘鐩存帴搴旂敤浜庤法鍩熺殑瀹夊叏鐜,鍥犺�涓嶈兘鐩存帴搴旂敤浜嶩adoop鍒嗗竷寮忓钩鍙般�

鏈枃鍒嗘瀽Hadoop浜戣绠楀钩鍙扮殑瀹夊叏闇�眰,鎻愬嚭涓�鍩轰簬韬唤鐨凜apability (ID-CAP), 鐒跺悗鎻愬嚭涓�鍩轰簬ID-CAP鐨凥adoop璁块棶鎺у埗鏂规銆�/p>

鍦ㄥ熀浜嶩adoop骞冲彴妗嗘灦寤虹珛鐨勬敮鎸佸绉熸埛鐨勪簯璁＄畻鏈嶅姟涓�鐢变簬姣忎釜鐢ㄦ埛閮藉彲浠ユ彁浜よ嚜宸辩殑MapReduce浣滀笟(job)鏉ヨ繍琛�閭ｄ箞鏈�熀鏈殑瀹夊叏闇�眰鏄疄鐜扮敤鎴疯璇佸拰璁块棶鎺у埗,骞朵笖瑕佹彁渚涗綔涓氳繍琛屾椂鐨勭浉浜掗殧绂�浠ラ槻姝綔涓氫箣闂寸浉浜掑共鎵�鏇磋闃叉鎭舵剰浣滀笟瀵瑰簳灞傚钩鍙板甫鏉ュ畨鍏ㄩ闄┿�

涓轰簡瀹炵幇杩欎竴瀹夊叏鐩爣,鏈枃鎻愬嚭鐨勬柟妗堝熀浜庡涓嬪亣璁�

1) 闆嗙兢鍒濆鍖栨椂鏄彲淇＄殑銆傚湪闆嗙兢鍒濆鍖栫殑绗竴澶�璁や负杩欎釜鐜鏄共鍑�殑,涓嶅瓨鍦ㄤ换浣曟敾鍑昏�銆�/p>

2) 鐢ㄦ埛鍙兘鎻愪氦鎭舵剰鐨凪apReduce job銆傛瘮濡�鎭舵剰鐨刯ob鍙互鍚戝唴閮ㄦ湇鍔�渚嬪, NameNode)鍙戦�浠绘剰鐨凴PC璇锋眰銆�/p>

3) 鏀诲嚮鑰呮棤娉曡幏鍙栧簳灞侺inux Kernel鐨剅oot鏉冮檺銆傝繖鎰忓懗鐫�鏀诲嚮鑰呬笉鑳界洃鍚綉缁�涓嶈兘绡℃敼缃戠粶鏁版嵁鍖�涓嶈兘璁块棶鏈湴鏂囦欢绯荤粺涓殑鏁忔劅鐩綍,鏃犳硶绡℃敼鐗╃悊鍐呭瓨,鏃犳硶淇敼瀹夊叏绛栫暐绛夈�

鍋囪3鐪嬭捣鏉ユ湁鐐逛笉鍒囧疄闄�鍥犱负Linux浣滀负涓�釜鍟嗙敤鎿嶄綔绯荤粺,涔熺粡甯告毚闇睰ernel鐨勪竴浜涙紡娲炲強鏀诲嚮鏂规硶銆備絾鏄�鍙互璁や负浜戞湇鍔℃彁渚涘晢鍙互缁勫悎澶氱Linux娌欑鎶�湳,閫氳繃瀹炵幇绾垫繁闃插尽绛栫暐鏉ユ弧瓒冲亣璁�銆�/p>

鍦ㄦ帴涓嬫潵鐨勬弿杩颁腑,灏嗙敤鍒扮殑绗﹀彿濡�span class="xref">琛�鎵�ず銆�/p>

鍩轰簬Capability鐨勮闂帶鍒舵柟妗堝湪鏂嘯1-2]宸叉湁鎻忚堪銆備竴涓狢apability鍙互鐪嬩綔鐢ㄦ埛璁块棶鏉冮檺鐨勯泦鍚堛�涓�釜缁忓吀鐨凜apability鍙互琛ㄧず涓哄涓嬬殑涓�釜浜屽厓缁�

(PermList,Random)

鍏朵腑:

1) PermList={(Object, Permissions)}銆傚浜庝竴涓粡鍏告枃浠剁郴缁熺殑PermList鏉ヨ, Object閫氬父涓烘爲鐘剁粨鏋�濡傜洰褰曟垨鏂囦欢璺緞銆侾ermissions涓篟ead, Write, eXecutable銆備妇渚嬫潵璇� PermList鐨勫�鍙互涓� {(鈥�home/admin/鈥� RW), (鈥�home/data/1.txt鈥� R), (鈥�usr/sbin/ping鈥� X)}銆傚浣曞畾涔変竴涓狾bject鐨勬牸寮忔槸涓庡叿浣撴湇鍔＄浉鍏崇殑,姣斿鍦℉adoop骞冲彴涓�鍙互浣跨敤(鈥渉dfs: /home/admin/鈥� RW)鏉ヨ〃绀哄厑璁歌鍐欒闂瓾DFS鏂囦欢绯荤粺涓殑 鈥�home/admin/鈥濈洰褰曞強璇ョ洰褰曚笅鐨勬墍鏈夊瓙鐩綍鍙婃枃浠�杩樺彲浠ヤ娇鐢�鈥渕r: /alice/job1鈥� R)鏉ヨ〃绀哄厑璁歌鍙朚apReduce鐢ㄦ埛alice鐨勪綔涓歫ob1鐨勮繍琛岀姸鎬併�

2) Random鏄竴涓敤浜庢姉浼�鏀诲嚮鐨勯殢鏈烘暟,鍏跺吀鍨嬪疄鐜版槸鏁板瓧绛惧悕(digital signature)鎴栨秷鎭璇佺爜(message authentication code)銆�/p>

鍏充簬Capability鐨勫巻鍙茬爺绌惰緝澶�sup>[3,4,5,6,7,8]銆侰apability璁块棶鎺у埗鐨勬樉钁楃壒寰佹槸鎵ц鏉冮檺妫�煡鏃舵�鑳介珮銆傚綋Capability闅忚姹備竴璧蜂紶閫掔粰Server鏃� Capability鍙互鍦ㄦ湰鍦拌繘琛岄獙璇�鑰屾棤闇�笌绗笁鏂逛氦浜�姣斿,涓嶉渶瑕佽闂换浣曟潈闄愭暟鎹簱)銆傝�鍩轰簬ACL璁块棶鎺у埗鍒楄〃鐨勬柟娉曞垯闇�璁块棶鏉冮檺涓績鏁版嵁搴�鎴栬�闇�鏄傝吹鐨勫垎甯冨紡缂撳瓨鏈嶅姟銆侰apability鐨勫彟涓�釜鐗瑰緛鏄兘鏇存湁鏁堝湴鏀寔鏈�皬鎺堟潈鍘熷垯,瀹冩槸鍙栧緱绯荤粺瀹夊叏鎬у拰绋冲畾鎬х殑鍩虹銆傚綋閮ㄥ垎瀹夊叏鍔熻兘閬彈鏀诲嚮鏃�璇ュ師鍒欒兘灏嗗畨鍏ㄩ闄╂帶鍒跺埌鏈�皬銆侰apability鐨勮繖浜涚壒鐐逛娇鍏舵洿閫傚悎浜庡垎甯冨紡璁＄畻鐜銆�/p>

鐒惰�,缁忓吀Capability绯荤粺鏈変竴涓富瑕佺殑涓嶈冻鈥斺�鎷ユ湁璁块棶鑳藉姏鍗虫嫢鏈夋巿鏉冭兘鍔涖�杩欏氨鏄,涓�釜Capability鍙互琚鍒跺苟浼犻�缁欏叾浠栧畨鍏ㄥ煙鐨勭敤鎴蜂娇鐢ㄣ�Boebert鍦ㄦ枃[7]涓寚鍑�缁忓吀鐨凜apability绯荤粺涓嶈兘瀹炴柦BLP妯″瀷涓殑 *-property[8], 鎴栬�璇存棤娉曡В鍐砪onfinement闂[9]銆傜敱浜庣粡鍏窩apability鎵�瓨鍦ㄧ殑瀹夊叏闂,瀹冧笉鑳借鐩存帴鐢ㄤ簬璺ㄥ煙鐨勫垎甯冨紡璁＄畻鐜銆�/p>

鍦℉adoop鍒嗗竷寮忕幆澧冧腑, Hadoop MapReduce瀹夊叏妗嗘灦浠ュ強HDFS鏈嶅姟杩愯鍦ㄩ珮瀹夊叏灞傜骇,鑰岀敤鎴锋彁浜ょ殑MapReduce job鍒欒繍琛屼簬浣庡畨鍏ㄥ眰绾с�杩欐剰鍛崇潃Hadoop鐜娑夊強鍒拌嚦灏�涓畨鍏ㄥ煙銆傚洜姝�缁忓吀鐨凜apability鏂规鐩存帴鐢ㄤ簬Hadoop鐜鏄笉瀹夊叏鐨勩�涓轰簡瑙ｅ喅杩欎釜闂,缁忓吀Capability鐨勮涔夐渶瑕佽鎵╁厖,姣斿澧炲姞涓讳綋韬唤淇℃伅,杩欏氨鏄笅鏂囨彁鍑虹殑鍩轰簬韬唤鐨凜apability鏂规銆�/p>

ID-CAP (identity-based capability)鏄粡鍏窩apability鐨勪竴绉嶆墿灞曘�涓庣粡鍏窩apability鐩告瘮, ID-CAP鍖呭惈浜嗘洿澶氱殑瀹夊叏灞炴�,濡傛寔鏈変汉韬唤,鎸佹湁浜虹殑鍏挜銆傚舰寮忓湴, ID-CAP鏄涓嬪睘鎬х殑涓�釜鏁板瓧绛惧悕:

ID-CAP=(ID,OwnerID,OwnerPK,Type,ExpTime,PermList)SK.

鍏朵腑,姣忛」鍚箟濡備笅:

1) ID: ID-CAP鐨処D銆�/p>

2) OwnerID: Capability鎸佹湁鑰呯殑鐢ㄦ埛ID銆�/p>

3) OwnerPK: Capability鎸佹湁鑰呯殑璁よ瘉鍏挜銆�/p>

4) Type: Capability鐨勭被鍨嬪畾涔�姣斿bearer|renewable銆�/p>

5) ExpTime: Capability鐨勮繃鏈熸椂闂淬�

ID-CAP鍙湅浣滀竴绉嶇壒娈婄殑鏀寔璁块棶鎺у埗灞炴�鐨勮韩浠借瘉涔︺�鍩轰簬ID-CAP, 楠岃瘉鑰呭彲浠ヨ幏寰楁寔鏈変汉鐨勫叕閽ュ強鍏惰闂潈闄愩�姝ゅ, ID-CAP鎸佹湁鑰呭彲浠ユ槸鐢ㄦ埛,涔熷彲浠ユ槸浣滀笟杩涚▼銆傚湪鏈枃鐨勬柟妗堣璁′腑, Capability鎸佹湁浜烘槸鎸囦綔涓氳繘绋嬨�

2.3.1 浜х敓ID-CAP

ID-CAP鍙兘鐢变簯璁＄畻骞冲彴鐨勫唴閮–A涓績绛惧彂銆傛鏃�鍐呴儴CA涓績鎻愪緵鐨勪粎浠呮槸涓�釜鏁板瓧绛惧悕鏈嶅姟銆傚唴閮–A涓績鍙互鎻愪緵涓�釜鏁板瓧绛惧悕鏈嶅姟鐨凙PI鎺ュ彛鍜岃闂帶鍒跺姛鑳�渚涘叾浠栨湇鍔″櫒璋冪敤銆�/p>

2.3.2 楠岃瘉ID-CAP

ID-CAP涓�埇涓烘秷鎭姹備腑鐨勫弬鏁颁箣涓��涓�釜Client璇锋眰璁块棶Server鏃�蹇呴』浣跨敤鏀寔ID-CAP鐨勬湇鍔℃帴鍙ｃ�褰揝erver鏀跺埌娑堟伅璇锋眰鏃�棣栧厛瑕佹鏌D-CAP鐨勬湁鏁堟�,姝ラ濡備笅:

1) 楠岃瘉鏁板瓧绛惧悕鐨勬湁鏁堟�;

2) 楠岃瘉鏈夋晥鏈�

3) 楠岃瘉Type.Bearer灞炴�鏄惁鏈夎璁剧疆,鑻ユ湁璁剧疆鍒欒繘鍏ヤ笅涓�; 鍚﹀垯,闇�楠岃瘉ID-CAP鎸佹湁浜虹殑韬唤: 鏍规嵁ID-CAP涓殑鍏挜OwnerPK鏉ヨ繘涓�楠岃瘉娑堟伅璇锋眰鐨勬暟瀛楃鍚嶇殑鏈夋晥鎬�

4) 鏍规嵁ID-CAP涓殑PermList鏉ュ垽鏂槸鍚﹀寘鍚湰娆¤姹傛墍闇�鐨勮闂潈闄�鑻ユ弧瓒�鍒欐潈闄愭鏌ラ�杩� 鍚﹀垯,鏉冮檺妫�煡澶辫触銆�/p>

2.3.3 鎾ら攢ID-CAP

ID-CAP鍙互閫氳繃CRL鎾ら攢鍒楄〃鏈哄埗琚挙閿�涔熷彲浠ユ敮鎸佽嚜鍔ㄦ挙閿��CRL鎾ら攢鍒楄〃鍙互鍙傝�PKI浣撶郴鐨勮瘉涔︽挙閿�柟娉曘�瀵逛簬鑷姩鎾ら攢鏉ヨ,褰揈xpTime鍒版潵鏃� ID-CAP灏嗚嚜鍔ㄥけ鏁堛�濡傛灉涓�釜浣滀笟鐨勮繍琛屾椂闂磋秴杩嘐xpTime, ID-CAP鐨凾ype.Renewable灞炴�闇�琚缃�褰揟ype.Renewable琚缃椂,鍦‥xpTime鍒版潵涔嬪墠,鍙互鎹㈠彇涓�釜鎷ユ湁鏇撮暱鐨勮繃鏈熸椂闂寸殑ID-CAP銆�/p>

涓嬮潰鍩轰簬ID-CAP, 鎻愬嚭涓�婊¤冻鏈�皬鎺堟潈鍘熷垯鐨凥adoop璁块棶鎺у埗鏂规,瀹冭兘涓虹敤鎴锋彁浜ょ殑姣忎釜job鍒嗛厤鏈�皬鏉冮檺,浠庤�鎻愪緵鏇村己鐨勫閿欒兘鍔涘拰绯荤粺瀹夊叏鎬с�

Hadoop浣滀负浜戣绠楁湇鍔＄殑涓�釜搴曞眰骞冲彴妗嗘灦,鏈枃鐨勮璁″熀浜庝互涓嬭�铏�

1) 鏈�皬鎺堟潈鍘熷垯銆傝繖涓�師鍒欐槸鎻愰珮鍒嗗竷寮忕郴缁熷畨鍏ㄦ�鍜岀ǔ瀹氭�鐨勬牳蹇冦�浠庡閿欏拰鎶靛埗鎭舵剰鏀诲嚮鐨勮搴︽潵鐪�瀹炴柦鏈�皬鏉冮檺鍙互鏈夋晥鐨勫姞寮烘暟鎹繚鎶ゃ�鍦℉adoop绯荤粺涓�鏈枃鐨勭洰鏍囨槸璁╂瘡涓�釜浣滀笟閮戒互鏈�皬鏉冮檺杩愯銆�/p>

2) 浣滀笟绾ч殧绂汇�鍦ㄥ師鐢烪adoop绯荤粺涓�涓�釜鐢ㄦ埛鎻愪氦鐨勬墍鏈変綔涓氶兘鍏锋湁鐩稿悓鐨勮闂潈闄�鑰屼笖鍙兘鍋氬埌鐢ㄦ埛绾у埆鐨勯殧绂�鑰屾棤娉曟弧瓒充綔涓氱骇鐨勯殧绂汇�閭ｄ箞,涓�釜浣滀笟濡傛灉閬彈鏀诲嚮鎴栭亣鍒伴敊璇�寰堝鏄撴畠鍙婅鐢ㄦ埛鐨勬墍鏈変綔涓氥�

3) 楂樻�鑳姐�鍦ㄤ簯璁＄畻绯荤粺涓�鎬ц兘寰�線鏄墍杩芥眰鐨勪竴涓牳蹇冩寚鏍囥�瑙ｄ緷璧栨槸鎻愰珮鎬ц兘鐨勪竴涓父鐢ㄦ柟娉�灏ゅ叾鏄珮棰戞搷浣滀竴瀹氫笉鑳戒緷璧栦簬鏌愪釜鍗曠偣銆傛瘮濡�鏉冮檺妫�煡鎿嶄綔灏辨槸涓�釜楂橀鎿嶄綔,灏辫鍋氬埌涓嶄緷璧栦簬鍗曠偣鏈嶅姟(姣斿,涓嶅簲璇ヤ笌鏉冮檺鏁版嵁搴撴湇鍔¤繘琛屼氦浜�銆�/p>

鍦ㄦ湰鏂囩殑鏂规涓�闄や簡MapReduce鍜孒DFS鏈嶅姟涔嬪,闇�鏈変竴涓唴閮–A鏈嶅姟鍜屼竴涓狟roker鏈嶅姟銆侰A鏈嶅姟涓昏鎻愪緵鍚勬湇鍔℃ā鍧楀強鐢ㄦ埛浣滀笟鐨勫瘑閽ラ鍙戝拰ID-CAP棰佸彂鏈嶅姟銆侭roker鏈嶅姟涓昏鎻愪緵鐢ㄦ埛鐨勬帴鍏ャ�

3.2.1 骞冲彴鍚姩

鍦ㄩ儴缃插惎鍔℉adoop闆嗙兢涔嬪墠,闇�纭繚CA鏈嶅姟姝ｅ父宸ヤ綔銆傞鍏堜负Hadoop鍚勪釜Server (eg, MapReduce, HDFS, ZooKeeper) 鐢宠Server PK/SK浠ュ強鍚勮嚜鐨処D-CAP銆傝嫢鏄涓�閮ㄧ讲鎴栭儴缃查�椤逛腑鏄惧紡瑕佹眰鏇存柊瀵嗛挜瀵瑰拰ID-CAP, 閭ｄ箞CA浼氭挙閿�erver涓婁竴娆′娇鐢ㄧ殑鏈夋晥ID-CAP, 骞朵负鍚勪釜Server鍒涘缓涓�釜闅忔満鐨凷erver PK/SK瀵嗛挜瀵� 鑻ユ槸閲嶅惎闆嗙兢骞舵棤闇�洿鏂板瘑閽ュ鍜孖D-CAP, 閭ｄ箞CA浼氱户缁繚鐣橲erver姝ｅ湪浣跨敤鐨勫瘑閽ュ鍜孖D-CAP銆傞儴缃叉垚鍔熷悗, MapReduce鐨凧obTracker鍜孴askTrackers閮戒細鑾峰緱鐩稿悓鐨処D-CAPT鍜�PKT,SKT), HDFS鐨凬ameNode鍜孌ataNodes鍒欎細鑾峰緱鐩稿悓鐨処D-CAPN鍜�PKN,SKN)銆�/p>

鐒跺悗,鐢宠Broker Server鐨凱K/SK鍜孖D-CAP, 鐢宠鏂瑰紡涓庡叾浠栨ā鍧楃被浼笺�閮ㄧ讲鎴愬姛鍚� Broker Server灏嗕細鑾峰緱鑷繁鐨処D-CAPB鍜�PKB,SKB)銆�/p>

3.2.2 浣滀笟鎻愪氦

Hadoop骞冲彴鏈嶅姟鍚姩涔嬪悗,鐢ㄦ埛灏卞彲浠ユ彁浜apReduce浣滀笟浜嗐�鍥�鎻忚堪浜嗕綔涓氭彁浜ょ殑娴佺▼銆�/p>

鍦�span class="xref">鍥�涓� Client浣嶄簬Hadoop骞冲彴鐨勫闈�鍏朵粬缁勪欢灞炰簬骞冲彴鐨勫唴閮ㄦā鍧椼�Broker鏄暣涓狧adoop骞冲彴鐨勫叆鍙�瀹冩墽琛岀敤鎴疯韩浠借璇佷笌鏉冮檺妫�煡閫昏緫銆侰A鏄暣涓狧adoop骞冲彴鐨勫瘑閽ョ鐞嗕腑蹇�璐熻矗瀵嗛挜鍙奍D-CAP鐨勯鍙戙�JobTracker鍜孴askTracker涓篐adoop MapReduce妗嗘灦鍐呯粍浠�璐熻矗鎵ц鐢ㄦ埛鎻愪氦鐨凪apReduce浣滀笟銆�/p>

涓嬮潰璇︾粏鎻忚堪鎵ц娴佺▼:

Step 1 Client閫氳繃澶栭儴璁よ瘉涔嬪悗, Client鍚態roker鎻愪氦浣滀笟璇锋眰,璇锋眰鍐呭鍖呮嫭:

(Client鈥檚 identity proof,JobExecutable,JobConfiguration)

鐢变簬Hadoop涓嶅仛鐢ㄦ埛绠＄悊涓庤璇� Client闇�杩涜澶栭儴璁よ瘉,姣斿Kerberos SSO璁よ瘉鏈嶅姟鐧诲綍,姝ゆ椂Client鑾峰緱鐨勮韩浠借瘉鏄庡氨鏄竴涓闂瓸roker鐨凨erberos ST銆傚綋Broker鏀跺埌璇锋眰鍚�棣栧厛楠岃瘉Client鐨勮璇佸嚟鎹�纭Client鐨勭敤鎴疯韩浠�鑾峰緱uid銆傚垎鏋愪綔涓氶厤缃枃浠禞obConfiguration, 鑾峰彇姝や綔涓氶渶瑕佽闂殑HDFS鏂囦欢璧勬簮鍙婂叾璁块棶鏉冮檺currentPermList銆傛煡璇㈢敤鎴锋潈闄愭暟鎹�纭繚currentPermList鏄鐢ㄦ埛鐨刾ermList鐨勪竴涓瓙闆�鍚﹀垯杩斿洖鏉冮檺妫�煡澶辫触鐨勫嚭閿欎俊鎭�

Step 2 Broker鍚慍A鐢宠棰佸彂璇ob鐨処D-CAP J銆傝璇锋眰鍐呭涓�

(ID-CAPB,(uid,currentPermList,Type,ExpTime)SK,B).

鍏朵腑ID-CAPB鏄疊roker鍦ㄥ垵濮嬪寲鏃惰幏鍙栫殑Capability, 鍙互鐢ㄤ簬璁块棶CA鍜孞obTracker鐨勯儴鍒咥PI銆係KB鏄疊roker鐨勭閽�鐢ㄤ簬鏁板瓧绛惧悕銆�/p>

Step 3 褰揅A鏀跺埌棰佸彂ID-CAPJ鐨勮姹傛椂,棣栧厛楠岃瘉璇锋眰鑰呯殑韬唤鏄惁涓築roker, 杩欐槸閫氳繃楠岃瘉璇锋眰鐨勬暟瀛楃鍚嶆潵瀹屾垚鐨勩�鐒跺悗楠岃瘉ID-CAPB鏉ユ鏌ヨ姹傝�鐨勮闂潈闄愩�楠岃瘉閫氳繃鍚�涓鸿Job浜х敓涓�釜鍏閽ュ(PKJ,SKJ), 鏍规嵁PKJ銆�uid銆�currentPermList銆�Type銆�ExpTime绛夎姹傚弬鏁�浜х敓濡備笅鐨処D-CAPJ:

ID-CAPJ=(ID,uid,PKJ,Type,ExpTime,currentPermList)SK.

鏈�悗, CA灏哠KJ鍜孖D-CAPJ杩斿洖缁橞roker銆�/p>

Step 4 鏀跺埌CA杩斿洖鐨凷KJ鍜孖D-CAPJ涔嬪悗, Borker灏嗕綔涓歋KJ鍜孖D-CAPJ娣诲姞鍒癑obConfiguration, 鍚慗obTracker鎻愪氦SubmitJob璇锋眰鏃堕渶瑕佷紶閫掑涓嬪弬鏁�

(ID-CAPB,(Request)SK,B).

JobTracker鏀跺埌璇锋眰鍚�棣栧厛楠岃瘉Request鐨勬暟瀛楃鍚�鐒跺悗楠岃瘉ID-CAPB鐨勬湁鏁堟�鍙婂叾璁块棶鏉冮檺銆傞獙璇侀�杩囧悗,浠嶳equest涓彁鍙朖obConfiguration, 鐒跺悗瑙ｆ瀽璇ob杩愯鏃舵墍闇�鐨凷KJ鍜孖D-CAPJ, 骞朵紶閫掔粰TaskTracker銆�/p>

3.2.3 浣滀笟杩涚▼璁块棶HDFS

TaskTracker鍚姩job宸ヤ綔杩涚▼,浼氫负宸ヤ綔杩涚▼閰嶇疆濂絊KJ鍜孖D-CAPJ銆傚綋job宸ヤ綔杩涚▼闇�璁块棶HDFS鏃�璁块棶鎺у埗鐨勬墽琛屾祦绋嬪鍥�鎵�ず銆�/p>

浣滀笟杩涚▼浠ヤ竴涓狧DFS Client瑙掕壊涓嶩DFS閫氫俊銆傚亣璁綜lient鎵ц鏂囦欢璇绘搷浣�姣斿璇绘枃浠秇dfs: /data/file1.txt, 涓嬮潰鏉ョ湅鍏蜂綋鐨勬墽琛屾祦绋嬨�

Step 1 Client鍚慛ameNode鎻愪氦OpenFileForRead璇锋眰鏃�闇�浼犻�濡備笅鍙傛暟:

(ID-CAPJ,(Request)SK,J).

Step 2 NameNode楠岃瘉Request鐨勬暟瀛楃鍚嶆槸鍚︽湁鏁�鐒跺悗楠岃瘉ID-CAPJ鏄惁鏈夊/data/file1.txt鐨勮鏉冮檺; 鑻ユ病鏈夎鏉冮檺,鍒欐嫆缁濄�楠岃瘉閫氳繃鍚� NameNode鍚慍lient鍙戦�鏂囦欢浣嶇疆淇℃伅,濡侱ataNode浣嶇疆, BlockInfo, 骞朵笖杩樺寘鍚竴涓闂瓺ataNode鐨則oken銆傛token涓篵earer绫诲瀷, DataNode鏃犻渶楠岃瘉鎸佹湁浜虹殑韬唤,浠讳綍Client鎸佹湁璇oken閮藉彲浠ヨ闂瓺ataNode銆備骇鐢焧oken鏂规硶濡備笅:

token=HMAC_SHA1(SKN,OwnerIDBlockInfoExpTime).

Step 3 Client鍚慏ataNode鍙戦�璇绘暟鎹姹�璇锋眰鍙傛暟闇�鍖呭惈姝ラ2涓帴鏀跺埌鐨則oken銆�/p>

Step 4 DataNode楠岃瘉token鐨勬湁鏁堟�,楠岃瘉閫氳繃鍚�杩斿洖鐩稿簲鐨勬暟鎹潡銆�/p>

3.2.4 Master-Slaves璁よ瘉

鍦℉adoop鍐呴儴, MapReduce鍜孒DFS閮藉彲浠ョ湅鍋氭槸涓�Master-Slaves缁撴瀯,姣斿JobTracker-TaskTrackers銆�NameNode-DataNodes銆傚湪绯荤粺鍚姩鏃� JobTracker涓嶵askTrackers閮借幏寰椾簡绉侀挜SKT, NameNode涓嶥ataNodes涔熼兘鑾峰緱浜嗙閽KN銆傚熀浜庡叡浜殑绉侀挜, Master涓嶴laves鎸夊涓嬫柟娉曡繘琛岃璇� 灏嗗叡浜殑绉侀挜浣滀负MasterKey, 浣跨敤HMAC_SHA1鏉ヨ绠楁秷鎭鍚�浠庤�瀹炵幇Master涓嶴laves涔嬮棿閫氫俊鐨勫彲璁よ瘉鎬с�

鍩轰簬Hadoop鐨勪簯璁＄畻骞冲彴涓昏鎻愪緵鏁版嵁鍒嗘瀽涓庡鐞嗘湇鍔°�Hadoop鐢ㄦ埛鐩存帴鎻愪氦鐨勬垨涓婂眰搴旂敤绋嬪簭鎻愪氦鐨凪apReduce浣滀笟閫氬父鍦ㄤ竴灞傛垨澶氬眰娌欑閲岃繍琛屻�鍦ㄧ2鑺傜殑濞佽儊妯″瀷涓�鍋囪娌欑鏈哄埗鏄畨鍏ㄧ殑,閭ｄ箞鎭舵剰鐨凪apReduce绋嬪簭灏辨棤娉曟帶鍒跺簳灞傛搷浣滅郴缁�鏃犳硶鐩戝惉鎴栫鏀归泦缇や腑浼犺緭鐨勭綉缁滄暟鎹寘銆�/p>

4.1.1 璁よ瘉鎬�/p>

1) Hadoop闆嗙兢鍐呴儴鍚勬ā鍧椾箣闂寸殑娑堟伅閫氫俊鏄彲璁よ瘉鐨勩�鍦╦ob鎻愪氦闃舵(瑙�.2.2鑺傛弿杩�, 姣忎竴涓秷鎭�淇￠兘鏈夎姹傝�鐨勬暟瀛楃鍚嶃�杩愯鍦ㄦ矙绠变腑鐨勬伓鎰忕▼搴忔棤娉曡幏鍙栧叾浠栦綔涓氳繘绋嬬殑绛惧悕瀵嗛挜,鎵�互鏃犳硶浼�娑堟伅璇锋眰鑰呯殑韬唤銆傚湪job杩愯闃舵(瑙�.2.3鑺傛弿杩�, 褰揅lient浣跨敤token鏉ヨ闂瓺ataNode鏃� DataNode鑳介獙璇乼oken鐨勭湡瀹炴�銆傜敱浜巘oken鏄疦ameNode浣跨敤HMAC璁＄畻鐨�鎵�娇鐢ㄧ殑鏄疍ataNode涓嶯ameNode涔嬮棿鎵�叡浜殑瀵嗛挜SKN, 杩愯鍦ㄦ矙绠变腑鐨勬伓鎰忎綔涓氳繘绋嬫棤娉曠獌鍙栧埌SKN, 鎵�互鏃犳硶浼�token銆�/p>

2) Hadoop闆嗙兢鍐呴儴鍚勬ā鍧楀唴閮ㄧ殑娑堟伅閫氫俊鏄彲璁よ瘉鐨勩�濡�.2.4鑺傛墍鎻忚堪, Master涓嶴laves涔嬮棿鐨勯�淇￠兘鏄熀浜庡叡浜殑瀵嗛挜鏉ヨ绠桯MAC鏉ュ畬鎴愭秷鎭璇佺殑,鎵�互鏃犳硶浼�娑堟伅璇锋眰鑰呯殑韬唤銆�/p>

4.1.2 浣滀笟绾х殑鏈�皬鏉冮檺鎺у埗

姣忎釜鐢ㄦ埛閮芥湁涓�釜鏈�ぇ鐨勬潈闄愰泦鍚�瀹冪敱Broker鏈嶅姟鎵�鐞嗐�鍦ㄥ師鐢烪adoop涓�鐢ㄦ埛鎻愪氦鐨勬瘡涓猨ob閮戒娇鐢ㄨ鐢ㄦ埛鐨勬渶澶ф潈闄愭潵鎵ц銆傚湪鏈枃鐨勬柟妗堜腑, Broker浼氭牴鎹瘡涓猨ob鐨勮緭鍏ュ拰杈撳嚭閰嶇疆鏉ヤ骇鐢熶竴涓甫鏈夋晥鏈熼檺鐨処D-CAP, 鍏惰闂潈闄愭槸鐢ㄦ埛鏉冮檺闆嗗悎鐨勪竴涓瓙闆�杩欎釜鏉冮檺瀛愰泦鍙弧瓒宠繍琛岃job鎵�渶瑕佺殑鏉冮檺,鑰屼笉浼氬鍑轰笌璇ヨ繍琛宩ob鏃犲叧鐨勬潈闄愩�

杩芥眰楂樻�鑳芥槸鏈柟妗堣璁＄殑涓�釜鍩烘湰鍘熷垯銆備负浜嗛獙璇佽繖涓�柟妗堢殑鏈夋晥鎬�鏈枃鍩轰簬Hadoop-1.0.2寮�彂浜嗕竴涓蹇佃瘉鏄庣郴缁熴�褰卞搷璇ユ柟妗堟�鑳界殑涓�釜鏈�ぇ鍥犵礌鏄瘑鐮佸嚱鏁版搷浣溿�鍦ㄦ墍鏈夌殑瀵嗙爜鍑芥暟鎿嶄綔涓�寮�攢鏈�ぇ鐨勬槸绛惧彂Capability鎿嶄綔,鍥犱负瀹冩秹鍙婂埌璁＄畻鏁板瓧绛惧悕鎿嶄綔銆備絾鏄敱浜嶩adoop骞冲彴涓昏鐢ㄤ簬绂荤嚎鏁版嵁澶勭悊涓氬姟,浣滀笟鐨勬彁浜ゅ苟涓嶆槸涓�釜棰戠箒鐨勬搷浣�鎵�互绛惧彂Capability涓嶄細鎴愪负鐡堕,鎵�互瀹冨苟涓嶆槸鎬ц兘浼樺寲鐨勭洰鏍囥�瀵嗙爜鎿嶄綔鐨勪紭鍖栫洰鏍囧簲璇ユ槸Capability楠岃瘉鎿嶄綔,鍥犱负杩欐槸涓�釜鐩稿綋棰戠箒鐨勬搷浣溿�閴翠簬姝�閫夋嫨鏁板瓧绛惧悕绠楁硶鏃�灏界璁＄畻ECC绛惧悕瑕佹瘮RSA绛惧悕楂樻晥寰堝,浣嗛獙璇丷SA绛惧悕鐨勬�鑳藉嵈瑕佹樉钁椾紭浜嶦CC绛惧悕銆傚洜姝ゆ湰鏂囬�鎷╀簡RSA鏁板瓧绛惧悕鏂规銆傚疄楠屼腑,閲囩敤浜� 024-bit RSA绛惧悕绠楁硶(鍙傝�PKCS#1 v2.1), 璇ユ柟妗堜腑閫夋嫨RSA鍏挜鎸囨暟 e=3, 绛惧悕楠岃瘉鎿嶄綔浠呴渶2娆℃ā涔樿繍绠椼�瀹為獙璇佹槑瀵嗙爜鎿嶄綔鎵�紩鍏ョ殑璁＄畻寮�攢鏄彲蹇界暐鐨勩�

4.2.1 闆嗙兢閰嶇疆

瀹為獙鐜浣跨敤1鍙扮墿鐞嗕富鏈哄拰9鍙癒VM铏氭嫙涓绘満銆侼ameNode, JobTracker鍏变韩鐗╃悊涓绘満, DataNodes鍜孴askTrackers浣跨敤铏氭嫙鏈恒�鐗╃悊涓绘満閰嶇疆: CPU 2.53 GHz, 24 Cores, 96 G鍐呭瓨, 2脳512 G 纾佺洏绌洪棿; 鍩轰簬Linux鐨凨VM(Kernel-based Virtual Machine)铏氭嫙鏈洪厤缃� CPU 2.53 GHz, 6 Cores, 10 G鍐呭瓨, 100 G纾佺洏绌洪棿銆傛澶�姣忎釜鑺傜偣涓婁娇鐢ㄧ殑Linux鐗堟湰涓篟HEL6.2, 2.6.32-220.el6.x86_64, Hadoop鐗堟湰涓篐adoop-1.0.2銆�/p>

4.2.2 Hadoop鍩哄噯娴嬭瘯

鑰冭檻鍒癏adoop骞冲彴涓昏鎻愪緵澶ц妯＄绾挎暟鎹鐞嗕笟鍔�鐢ㄦ埛涓�埇涓昏鍏虫敞鐨勫嚑涓熀鍑嗘祴璇曞寘鎷� HDFS璇诲啓娴嬭瘯銆佹帓搴忔祴璇曞拰MapReduce杩炵画鎬ф祴璇曘�涓嬮潰灏辫繖3涓熀鍑嗘祴璇曞仛浜嗘�鑳藉姣斻�

1) HDFS璇诲啓娴嬭瘯

鎵ц璇绘祴璇曞拰鍐欐祴璇曠殑宸ュ叿濡備笅:

$ hadoop jar hadoop-test-1.0.2.jar TestDFSIO -read -nrFiles 10 -fileSize 1 000.

$ hadoop jar hadoop-test-1.0.2.jar TestDFSIO -write -nrFiles 10 -fileSize 1 000.

鍩轰簬鍘熺敓Hadoop鍜屽熀浜庢湰鏂囨彁鍑虹殑ID-CAP鏂规鐨勫悶鍚愰噺鍜屽钩鍧嘔O鐜囩殑娴嬭瘯缁撴灉濡�span class="xref">琛�鎵�ず銆傚鏄撶湅鍑� ID-CAP鏂规骞舵病鏈夊鑷村悶鍚愰噺鍜屽钩鍧嘔O鐜囩殑鏄捐憲闄嶄綆銆�/p>

2) 鎺掑簭娴嬭瘯

姣忎釜鑺傜偣杩愯10涓猰ap浠诲姟,浜х敓10 GB鐨勯殢鏈轰簩杩涘埗鏁版嵁銆傛祴璇曞伐鍏峰涓�

$ hadoop jar hadoop-examples-1.0.2.jar randomwriter /benchmarks/random-data.

鎺掑簭鐨勬祴璇曞伐鍏峰涓�

$ hadoop jar hadoop-examples-1.0.2.jar sort /benchmarks/random-data /benchmarks/sorted-data.

鎺掑簭姝ｇ‘鎬ф鏌ョ殑娴嬭瘯宸ュ叿濡備笅:

$ hadoop jar hadoop-test-1.0.2.jar testmapredsort -sortInput /benchmarks/random-data -sortOutput /benchmarks/sorted-data.

鍩轰簬鍘熺敓Hadoop鍜屽熀浜庢湰鏂囨彁鍑虹殑ID-CAP鏂规鍦ㄨ繖3涓楠ょ殑杩愯鏃堕棿瀵规瘮缁撴灉濡�span class="xref">琛�鎵�ず銆傛祴璇曠粨鏋滄樉绀� ID-CAP鏂规瀵规帓搴忔祴璇曟病鏈夋槑鏄剧殑鎬ц兘涓嬮檷銆�/p>

3) MapReduce杩炵画鎬ф祴璇�/p>

姣忎釜job澶勭悊10 000琛屾枃鏈枃浠躲� 6涓猰ap銆�2涓猺educe銆�鍙敹闆嗘暟鎹�鍒嗗埆杩愯100娆�娴嬭瘯宸ュ叿濡備笅:

$ hadoop jar hadoop-test-1.0.2.jar mrbench -numRuns 100 -maps 6 -reduces 2 -inputLines 10 000.

鍩轰簬鍘熺敓Hadoop鍜屽熀浜庢湰鏂囨彁鍑虹殑ID-CAP鏂规鐨勫钩鍧囨墽琛屾椂闂村姣旂粨鏋滃琛�鎵�ず銆傛祴璇曠粨鏋滄樉绀� ID-CAP鏂规瀵筂apReduce杩炵画鎬ф祴璇曞苟娌℃湁鏄庢樉鐨勬�鑳戒笅闄嶃�

Hadoop浜戣绠楀钩鍙板畨鍏ㄦ棤娉曟弧瓒冲紑鏀剧幆澧冧笅鐨勫绉熸埛瀹夊叏闇�眰銆傛湰鏂囬噰鐢ㄤ簡鍒嗗竷寮忕郴缁熻璁＄殑鏈�皬鎺堟潈鍘熷垯,璁捐浜嗕竴绉嶅熀浜庤韩浠界殑Capability (ID-CAP), 骞舵彁鍑哄熀浜嶪D-CAP鐨凥adoop璁块棶鎺у埗鏂规銆傚疄璺佃〃鏄� 璇ユ柟妗堣兘鏈夋晥鍦板疄鐜板湪Hadoop骞冲彴涓婂疄鏂芥渶灏忔巿鏉冨師鍒�骞舵敮鎸佸钩鍙板唴閮ㄧ浉浜掍緷璧栫殑鍚勬ā鍧椾箣闂寸殑韬唤璁よ瘉,鑳芥湁鏁堟彁楂楬adoop骞冲彴鐨勬暣浣撳畨鍏ㄦ�銆�/p>